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Abstract 

The Clopper-Pearson confidence interval has ever been documented as an exact approach in 
some statistics htcrature. More recently, such approach of interval estimation has been introduced 
to probabilistic control theory and has been referred as non-conscrvativc in control community. 
In this note, we clarify the fact that the so-called exact approach is actually conservative. In 
particular, we derive analytic results demonstrating the extent of conservatism in the context 
of probabilistic robustness analysis. This investigation encourages seeking better methods of 
confidence interval construction for robust control purpose. 

1 Introduction 

Ever since Stengel and Ray originated the concept of stochastic robustness, there has been growing 
interest in developing probabilistic methods for robust control. Significant contributions have been 
madeby a number of researchers (see, e.g., [Il[l[3lll[5],[ini[IIl[l2l[l3l[Il[I3[l6l[I7] and the reference 
therein). A fundamental problem in the area of probabilistic robustness analysis is to estimate the 
probabihty that a certain robustness requirement is guaranteed for an uncertain dynamic system. 
The estimation of such probability rehes essentiahy on Monte Carlo simulation. When an estimate 
of the probability is obtained from i.i.d. observations of fixed sample size, an important concern 
is how accurate this estimate is. To be useful, a numerical method must include a basis for error 
assessment. The Monte Carlo method is no exception. Stengel and Ray |12l [H] first introduced 
the Clopper-Pearson confidence interval to evaluate the accuracy of estimation in the context of 
robustness analysis. In their works, such approach has been considered as non-conservative. 

In this note, we would like to clarify the fact that the Clopper-Pearson confidence interval is 
conservative. The erroneous understanding of the confidence interval is not due to researchers in 
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control area. Historically, such erroneous concept can be traced back to some statistical literature. 
The Clopper-Pearson confidence interval was usually referred as the "exact" confidence interval. It 
has been documented as a non-conservative approach in some statistics literature (see, for example. 
Page 697-698 of [9] and Page 95-103 of [8]). Although the so-called exact confidence interval was 
proposed by Clopper and Pearson in 1934, its rigorous probabilistic implication was not made clear 
until 1958 by Cluniess-Ross [7]. Interestingly, it was proved in [7] that such "exact" approach is 
actually conservative. 

Since the confidence coefficient is directly related to the risk and safety of control systems in the 
context of probabilistic design and analysis, a clear understanding of the conservatism of confidence 
interval construction will help making the tradeoff between the risk and performance enhancement. 
To this purpose, we investigate the conservatism of the confidence interval. We obtain analytic results 
indicating that, in the scenario of rare events (especially in the context of probabilistic robustness 
analysis), the conservatism is not trivial and better methods of confidence construction should be 
sought. 



2 Binomial Confidence Interval 

Let the probability space be denoted as (O, F, P) where ^l, F, P are the sample space, the algebra 
of events and the probability measure respectively. Let X be a Bernoulli random variable with 
distribution Pr{X = 1} = P^, Y>t{X = 0} = 1 - Px where P.y € (0, 1). Let the sample size N 
and confidence parameter 5 € (0, 1) be fixed. We refer an observation with value 1 as a successful 
trial. Let K denote the number of successful trials during the N i.i.d. sampling experiments. Let 
k = K{lo) where (j is a sample point in the sample space VL. 

The classic Clopper-Pearson lower confidence limit L]\f^k,5 and upper confidence limit U]\f,k,5 are 
given respectively by 

' if it = 

p if A; > 



LN,k,S '■ 
and 



U 



N,k,S 



) 1 if /c = TV 
[p iik<N 

where p G (0, 1) is the solution of equation Yl'jZo {^)p'{^ ~ v)^~^ ~ ^ ~ i ^^"^ P ^ (*-*' ^) 
solution of equation Yl^j=o ( ~ P)^~'' = f- Define random variable L : ^ [0, 1] by L{uj) := 
Ln,K{lu),5 Vcj € and random variable U : Q ^ [0,1] by C/(lj) := Ui^^k{uj),5 G ^- Then the 
random interval [L, U] is referred as the classic Clopper-Pearson confidence interval. Its probabilistic 
implication was quite often erroneously interpreted as 

Pr{L <¥x <U] = 1- 5 
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or 

Pr{L <Fx <U} = 1-6. 
However, it was proved by Cluniess-Ross [7] in 1958 that 

Pr{L <Fx <U} > 1- 5 

and 

Pr{L <Fx <U}>l-6. 

These inequalities have been demonstrated by numerical experiments reported in the literature. For 
a better understanding of the conservatism, especially in the context of probabilistic robustness 
analysis, we shall investigate analytically how conservative the Clopper-Pearson interval can be. 

3 How Conservative? 

We refer the exact value of Pr{L < Fx < U} as the coverage probability. We have the following 
results with regard to the conservatism of the Clopper-Pearson confidence interval. 

Theorem 1 Let N, 6 be fixed. If Fx or 1 — Fx is less than 1 — (|)^, then the coverage probability 
will be at least 1 — |- Moreover, (|) ^ < Fx < 1 — (f ) ^ i then the coverage probability is 1. 

Proof. For the simplicity of notation, define S{N,k,x) := X]^=o {'^)'^''(^ — x)^~^ for x G (0,1). 
Notice that Px > (|)^ if and only if 

S{N, N -l,Fx) = l-Fx'' <l-- 

and observe that, for fixed x € (0, 1), S{N,k,x) increases monotonically with respect to k, we have 
that S{N, iV - 1, Px) < 1 - I if and only if 

S{N,k -l,Fx) < 1-^ yke {!,■■■ ,N}. 

By Lemma (3.8 a) on page 277 of [3, we have that S{N,k — l,x) decreases monotonically with 
respect to x S (0, 1). Therefore, Lj^^^^^ < Fx for all /c G {1, • • • , N}. Recall that L^^^^g = < Fx, 
we thus have 

Px ^{^y ^ ML < = 1. (1) 

Similarly, we can show that 

rx<l-(^y ^ Pr{Px <U} = 1. (2) 
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Figure 1: Conservatism of Clopper-Pearson Confidence Interval. Probability Bound = 1 — (f)^- 
Plot A corresponds to 6 = 0.001, plot B corresponds to 5 = 0.01, plot C corresponds to 6 = 0.05. 

For the case that ¥x > (|) by ([T]) and Bonferoni's inequality 

Pr{L <¥x <U} > Pr{L < ¥x} + Pr{Px < U} - 1 
= Pr{Px < U} 

> ^4 

Similarly, for the other case that Fx < 1 - (f)'^ we can show that Pr{L <¥x <U} > 1 - ^. Thus 
the first statement is proved. 

Finally, the proof of second statement is completed by making use of Bonferoni's inequality and 
inequalities ^ and ([2]). 

□ 

Figure [1] shows bounds of binomial proportion for which the true coverage probability of the 
Clopper-Pearson confidence interval exceeds the prescribed confidence level by at least |. It can be 
seen that the conservatism is common in the scenario of studying rare events by the Monte Carlo 
method. For example, when constructing a 99% confidence interval for the event of instability based 
on 10, 000 Monte Carlo simulations, the true confidence level will be at least 99.5% if the probability 
of instability is smaller than 0.0005 (This number is the vertical coordinate of the point in plot B 
with horizontal coordinate 10,000). 
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4 Conclusion 



It is demonstrated that the Clopper-Pearson confidence interval is rather conservative for rare events. 
The coverage probabihty can easily exceed the specified confidence level by at least | and can be 
100%. Robustness issues such as instability and performance violation are normally interpreted as 
such rare events. Although the confidence parameter 5 is usually a small number, the impact can 
be enormous due to its particular connection to the stability and performance of control systems 
in the probabilistic robust control framework. Our investigation suggests seeking better methods of 
confidence construction which are rigorous and less conservative. 
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